Matrix similarity transformation